Photography and moviemaking have become staples in modern society. A photograph is typically a single image frame of a real scene. Movies can be described as a series of frames that together form what appears to the human eye to be a continuously moving image. Both photographs and movies are now found in both physical and digital formats.
In recent years, advances in technology have allowed creation of entirely three dimensional worlds. 3D graphic systems are able to produce an image on a two-dimensional screen of a display in such a manner that the image simulates three-dimensional effects. In such 3D systems, the surface of a 3D object to be represented is separated into a plurality of polygonal surfaces having various arbitrary shapes. Picture data representing the polygonal areas of the 3D object are successively stored in a frame memory having memory locations corresponding to positions on a display screen to accumulate picture data which, when supplied to the display, reconstruct an image which appears to be three-dimensional.
In such 3D systems the data representing each of the polygonal surfaces must be transformed in order to represent three-dimensional effects such as rotation of the object they represent. In 3D systems the image data for the various polygonal surfaces are produced in succession based on data indicating the depth of each polygonal surface from the plane of the display screen. Conventional 3D systems produce image data representing the polygonal surfaces of an object such that surfaces which cannot be seen from the point of view when displayed are produced first and stored in a display memory and the remaining data representing the polygonal surfaces are successively produced in order according to their depth from the screen. Consequently, image data representing a polygonal surface at the front of the object cover over the image data of reverse surfaces which previously were produced and stored. It is necessary, therefore, to include data indicating the depth of each polygonal surface (referred to as “Z data”) and the order in which the data representing the polygons are produced is determined by reference to such Z data. In the conventional 3D systems a Z buffer is provided to store the Z data in pixel units and the stored Z data are compared to determine a display preference.
In conventional 3D systems, an effect of the environment on 3D objects is relatively easy to compute, as the environment also has Z data assigned to it, in addition to X (horizontal) and Y (vertical) data. However, where the environment does not have Z data associated with it, and is thus by definition a 2D environment, effects of the 2D environment on 3D objects such as reflection, refraction, shadows, etc. have heretofore not been readily and accurately rendered.
It has also been proposed in the prior art to implement a system wherein 2D image data would be produced by means of a conventional 2D system and three-dimensional image data would be produced by means of a conventional 3D system independently of the 2D system. The 2D image data and the 3D image data which have been produced independently are then added upon conversion into a video signal to be supplied to a video display device. However, this system too fails to allow accurate rendering of environmental effects on a 3D object.
Additionally, both methods simply overlay the 3D object over the 2D background image.
Consider, for example, movies which add 3D objects to a background image of a real scene. Current methods render the 3D object in a 3D renderer, and composite the 3D image on the 2D frames of a film. Then, artists must go back and, frame by frame, manually draw shadows and reflections on the 3D object. This is a very time consuming and thus expensive job, considering that a typical movie runs at about 30 frames per second.
If the 3D object is supposed to be positioned behind something on the frame, present systems require that a user manually create an image mask that is exactly the same size and shape as the 2D object to be shown in front of the 3D object.
Another mask, a shadow mask, is created by hand for the shadowing created by or cast onto the 3D object. Shadowing is currently performed by dimming the image, which is not an accurate representation of a shadow. Rather, the dimming appears more like a fuzzy area rather than an accurate representation of how the shadow will be cast. The typical method is to manually hand-draw a shadow mask for each frame by using ADOBE® PHOTOSHOP® or other manual graphics program.
If the designer further wants reflections in the scene, artists are called upon to make reflection maps. These are texture maps that go on the 3D model. This requires the artists to estimate what the scene around the 3D object looks like, and map this onto the 3D model. One problem is that the reflections do not look realistic, particularly on rounded or angled surfaces. Heretofore methods have not been able to accurately create the natural deformation of the reflection due to curvature of the reflecting surface. Particularly, human artists find it very difficult to conceptualize this deformation and create it in the reflection in the composite image.
Again, the state of the art is to manually perform all of these functions frame by frame, as the 2D image data does not have Z data assigned to it.
Additionally, if a surface of the 3D object is partially transparent, artists merely shade the pixels of the 2D image that would be viewable through the transparent portion of the 3D object. However, most transparent surfaces are refractive. Current methods do not account for refractive distortion.
What is therefore needed is a way to automatically perform not only rendering of a 3D image in a 2D scene, but also to add realistic shadowing, reflection, refraction, transparency and other effects automatically. This would save an immense amount of man-hours when generating animations, as the role of artists could then be greatly reduced and even eliminated. Such a solution would also reduce the inherent flaws in the effects heretofore manually created by human artists.